The present invention pertains to error detection and correction in stored data and pertains particularly to targeted data protection.
Stored data are subject to corruption through many mechanisms, such as user misuse (e.g., unintended file deletions), malicious attack (e.g. computer virus or worm attacks), system failures (e.g. magnetic tape damage by a tape drive or disk drive head crashes), and natural physical processes (e.g. cosmic ray bombardment). Various methods are employed to ensure data integrity, or more accurately, to control the probability of data corruption for storage media. These methods are typically combined to achieve a satisfactory level of data integrity.
User behavior is an important component in system data integrity. Regularly scheduled data storage backups allow falling back to a previously known, valid data state when catastrophic data corruption occurs. Also, the chance of data corruption can be reduced by installing computer virus protection systems and keeping them up to date, and by replacing storage devices when they begin to demonstrate unreliability.
Different storage media have differing physical error rates. More sensitive or important data may be stored on more reliable storage media. For example, data stored on CD-R discs are less likely to be corrupted by strong magnetic fields than are data stored on floppy discs. Some storage systems can recognize the deterioration of portions of the media and shift data automatically to more reliable locations. An example is a hard drive magnetic storage system that partitions the storage area into sectors. When the hard drive system detects that a sector is deteriorating, it marks that sector as “bad” and no longer uses it to store data.
Finally, the effective error rate for a particular storage medium can be reduced through various forms of automatic data redundancies. Redundant array of inexpensive disks (RAID) storage systems duplicate data across multiple discs to protect data against a single disc failure. Storage systems typically use error control codes (ECC) to encode stored data with extra bits to allow the systems to detect and correct for storage bit errors when reading the stored data.
Typically, storage systems are designed such that a common storage medium and error correction scheme is used for all of the data stored. An example is the magnetic storage disk drive. During the design phase, engineers decide how much storage space to dedicate to redundant information (overhead), and how much to dedicate to data. This decision is driven by anticipated corruption rates, expected characteristics of corrupted data, and customer requirements.
In certain applications (e.g., storage of compressed audio or image data), specific parts of the stored data are more sensitive to corruption than are other areas. Sensitivity, in this case, is quantified by measuring the acceptability of reproduced sound or images as judged by end users. For example, a single-bit error in one place in the compressed image data may change the intensity of a one pixel by an amount so small that the change is visually imperceptible. A single-bit error in another part of the data might severely corrupt one half of the pixels in a row of data, resulting in an unacceptable artifact.
The typical course of action, during design of the system, is to store enough redundant information so at to be able to sufficiently insulate the most sensitive portions of the data from errors. This “overprotects” the rest of the stored data, resulting in sub-optimal storage efficiency. Alternatively, the amount of redundant information might be optimized for the less sensitive portions of data, resulting in greater customer dissatisfaction.